{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfcSMZEIFA_l"
   },
   "source": [
    "# ML with TensorFlow Extended (TFX) -- Part 3\n",
    "The puprpose of this tutorial is to show how to do end-to-end ML with TFX libraries on Google Cloud Platform. This tutorial covers:\n",
    "1. Data analysis and schema generation with **TF Data Validation**.\n",
    "2. Data preprocessing with **TF Transform**.\n",
    "3. Model training with **TF Estimator**.\n",
    "4. Model evaluation with **TF Model Analysis**.\n",
    "\n",
    "This notebook has been tested in Jupyter on the Deep Learning VM.\n",
    "\n",
    "## 0. Setup Python and Cloud environment\n",
    "\n",
    "Install libraries we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JkkjO6RyH5UP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -q --upgrade tensorflow_data_validation tensorflow_model_analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.local/lib/python3.5/site-packages/apache_beam/__init__.py:84: UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully supported. You may encounter buggy behavior or missing features.\n",
      "  'Running the Apache Beam SDK on Python 3 is not yet fully supported. '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tornado version: 6.0.2\n",
      "Python version: 3.5.3\n",
      "TF version: 1.13.1\n",
      "TFT version: 0.13.0\n",
      "TFDV version: 0.13.1\n",
      "Apache Beam version: 2.11.0\n"
     ]
    }
   ],
   "source": [
    "import apache_beam as beam\n",
    "import platform\n",
    "import tensorflow as tf\n",
    "import tensorflow_data_validation as tfdv\n",
    "import tensorflow_transform as tft\n",
    "import tornado\n",
    "\n",
    "print('tornado version: {}'.format(tornado.version))\n",
    "print('Python version: {}'.format(platform.python_version()))\n",
    "print('TF version: {}'.format(tf.__version__))\n",
    "print('TFT version: {}'.format(tft.__version__))\n",
    "print('TFDV version: {}'.format(tfdv.__version__))\n",
    "print('Apache Beam version: {}'.format(beam.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = 'cloud-training-demos'    # Replace with your PROJECT\n",
    "BUCKET = 'cloud-training-demos-ml'  # Replace with your BUCKET\n",
    "REGION = 'us-central1'              # Choose an available region for Cloud MLE\n",
    "\n",
    "import os\n",
    "\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "Updated property [compute/region].\n",
      "Updated property [ml_engine/local_python].\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION\n",
    "\n",
    "## ensure we predict locally with our current Python environment\n",
    "gcloud config set ml_engine/local_python `which python`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img valign=\"middle\" src=\"images/tfx.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l9u699PmHJXU"
   },
   "source": [
    "### Flights dataset\n",
    "\n",
    "We'll use the flights dataset from the book [Data Science on Google Cloud Platform](http://shop.oreilly.com/product/0636920057628.do)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "ksuSTsysHfZV",
    "outputId": "87adfbf0-be77-4d81-9162-5a2f9feffd90"
   },
   "outputs": [],
   "source": [
    "DATA_BUCKET = \"gs://cloud-training-demos/flights/chapter8/output/\"\n",
    "TRAIN_DATA_PATTERN = DATA_BUCKET + \"train*\"\n",
    "EVAL_DATA_PATTERN = DATA_BUCKET + \"test*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "CSV_COLUMNS = ('ontime,dep_delay,taxiout,distance,avg_dep_delay,avg_arr_delay' + \n",
    "               ',carrier,dep_lat,dep_lon,arr_lat,arr_lon,origin,dest').split(',')\n",
    "TARGET_FEATURE_NAME = 'ontime'\n",
    "DEFAULTS     = [[0.0],[0.0],[0.0],[0.0],[0.0],[0.0],\\\n",
    "                ['na'],[0.0],[0.0],[0.0],[0.0],['na'],['na']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-qNjgoalG0Xu"
   },
   "source": [
    "## 3. Model Training\n",
    "For training the model, we use [TF Estimators](https://www.tensorflow.org/guide/estimators) APIs to train a premade DNNClassifier. We perform the following:\n",
    "1. Load the **transform schema**\n",
    "2. Use the transform schema to parse TFRecords in **input_fn**\n",
    "3. Use the transform schema to create **feature columns**\n",
    "4. Create a premade **DNNClassifier**\n",
    "5. **Train** the model\n",
    "6. Implement the **serving_input_fn** and apply the **transform logic**\n",
    "7. **Export** and test the saved model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U4eLEbV42XEz"
   },
   "source": [
    "### 3.1 Load transform output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/flights/tfx/transform/\n",
      "gs://cloud-training-demos-ml/flights/tfx/transform/transform_fn/\n",
      "gs://cloud-training-demos-ml/flights/tfx/transform/transformed_metadata/\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00000-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00001-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00002-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00003-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00004-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00005-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00006-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/eval-00007-of-00008.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00000-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00001-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00002-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00003-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00004-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00005-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00006-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00007-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00008-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00009-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00010-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00011-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00012-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00013-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00014-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00015-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00016-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00017-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00018-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00019-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00020-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00021-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00022-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00023-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00024-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00025-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00026-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00027-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00028-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00029-of-00031.tfrecords\n",
      "gs://cloud-training-demos-ml/flights/tfx/transformed/train-00030-of-00031.tfrecords\n"
     ]
    }
   ],
   "source": [
    "PREPROC_OUTPUT_DIR = 'gs://{}/flights/tfx'.format(BUCKET)  # from 02_transform.ipynb\n",
    "TRANSFORM_ARTIFACTS_DIR = os.path.join(PREPROC_OUTPUT_DIR,'transform')\n",
    "TRANSFORMED_DATA_DIR = os.path.join(PREPROC_OUTPUT_DIR,'transformed')\n",
    "!gcloud storage ls $TRANSFORM_ARTIFACTS_DIR\n",    "!gcloud storage ls $TRANSFORMED_DATA_DIR"   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HR_d4-kBG0hI"
   },
   "outputs": [],
   "source": [
    "transform_output = tft.TFTransformOutput(TRANSFORM_ARTIFACTS_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bxt31_6M3dbl"
   },
   "source": [
    "### 3.2 TFRecords Input Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EQqf9dGF3dEe"
   },
   "outputs": [],
   "source": [
    "def make_input_fn(tfrecords_files, \n",
    "  batch_size, num_epochs=1, shuffle=False):\n",
    "\n",
    "  def input_fn():\n",
    "    dataset = tf.data.experimental.make_batched_features_dataset(\n",
    "      file_pattern=tfrecords_files,\n",
    "      batch_size=batch_size,\n",
    "      features=transform_output.transformed_feature_spec(),\n",
    "      label_key=TARGET_FEATURE_NAME,\n",
    "      reader=tf.data.TFRecordDataset,\n",
    "      num_epochs=num_epochs,\n",
    "      shuffle=shuffle\n",
    "    )\n",
    "    return dataset\n",
    "\n",
    "  return input_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<DatasetV1Adapter shapes: ({avg_dep_delay_scaled: (?, 1), distance_bucketized: (?, 1), avg_arr_delay_scaled: (?, 1), dep_lat_scaled: (?, 1), origin_integerized: (?, 1), distance_scaled: (?, 1), taxiout_scaled: (?, 1), arr_lon_scaled: (?, 1), dep_delay_scaled: (?, 1), dest_integerized: (?, 1), arr_lat_scaled: (?, 1), dep_lon_scaled: (?, 1), carrier_integerized: (?, 1)}, (?, 1)), types: ({avg_dep_delay_scaled: tf.float32, distance_bucketized: tf.int64, avg_arr_delay_scaled: tf.float32, dep_lat_scaled: tf.float32, origin_integerized: tf.int64, distance_scaled: tf.float32, taxiout_scaled: tf.float32, arr_lon_scaled: tf.float32, dep_delay_scaled: tf.float32, dest_integerized: tf.int64, arr_lat_scaled: tf.float32, dep_lon_scaled: tf.float32, carrier_integerized: tf.int64}, tf.float32)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_input_fn(TRANSFORMED_DATA_DIR+'/train*.tfrecords', 2, shuffle=False)()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vgQtTPiN4Td0"
   },
   "source": [
    "### 3.3 Create feature columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XzP3BUnU4bE5"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def create_feature_columns():\n",
    "\n",
    "  feature_columns = []\n",
    "  transformed_features = transform_output.transformed_metadata.schema._schema_proto.feature\n",
    "\n",
    "  for feature in transformed_features:\n",
    "\n",
    "    if feature.name in [TARGET_FEATURE_NAME]:\n",
    "      continue\n",
    "\n",
    "    if hasattr(feature, 'int_domain') and feature.int_domain.is_categorical:\n",
    "      vocab_size = feature.int_domain.max + 1\n",
    "      feature_columns.append(\n",
    "        tf.feature_column.embedding_column(\n",
    "          tf.feature_column.categorical_column_with_identity(\n",
    "            feature.name, num_buckets=vocab_size),\n",
    "            dimension = int(math.sqrt(vocab_size))))\n",
    "    else:\n",
    "      feature_columns.append(\n",
    "        tf.feature_column.numeric_column(feature.name))\n",
    "\n",
    "  return feature_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "wAttQuXh-ZXR",
    "outputId": "4f998a45-7c41-460c-8d66-e59bf18a507b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NumericColumn(key='arr_lat_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='arr_lon_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='avg_arr_delay_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='avg_dep_delay_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " EmbeddingColumn(categorical_column=IdentityCategoricalColumn(key='carrier_integerized', number_buckets=14, default_value=None), dimension=3, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x7f02e1c63eb8>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True),\n",
       " NumericColumn(key='dep_delay_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='dep_lat_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='dep_lon_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " EmbeddingColumn(categorical_column=IdentityCategoricalColumn(key='dest_integerized', number_buckets=322, default_value=None), dimension=17, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x7f02e1c63c88>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True),\n",
       " EmbeddingColumn(categorical_column=IdentityCategoricalColumn(key='distance_bucketized', number_buckets=5, default_value=None), dimension=2, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x7f02a4dff4e0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True),\n",
       " NumericColumn(key='distance_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " EmbeddingColumn(categorical_column=IdentityCategoricalColumn(key='origin_integerized', number_buckets=322, default_value=None), dimension=17, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x7f02a4dff438>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, max_norm=None, trainable=True),\n",
       " NumericColumn(key='taxiout_scaled', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_feature_columns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Cr4iLJtK4bgv"
   },
   "source": [
    "### 3.4 Instantiate and Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IKAGwUgP4Tkf"
   },
   "outputs": [],
   "source": [
    "def create_estimator(params, run_config):\n",
    "    \n",
    "  feature_columns = create_feature_columns()\n",
    "\n",
    "  estimator = tf.estimator.DNNClassifier(\n",
    "    #weight_column=WEIGHT_COLUMN_NAME,\n",
    "    #label_vocabulary=TARGET_LABELS,\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=params.hidden_units,\n",
    "    config=run_config\n",
    "  )\n",
    "\n",
    "  return estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iZIyscv84nXF"
   },
   "source": [
    "### 3.5 Implement train and evaluate experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e3zLhoZj4nd4"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def run_experiment(estimator, params, run_config, resume=False):\n",
    "  \n",
    "  tf.logging.set_verbosity(tf.logging.INFO)\n",
    "\n",
    "  if not resume: \n",
    "    if tf.gfile.Exists(run_config.model_dir):\n",
    "      print(\"Removing previous artifacts...\")\n",
    "      tf.gfile.DeleteRecursively(run_config.model_dir)\n",
    "  else:\n",
    "    print(\"Resuming training...\")\n",
    "\n",
    "  train_spec = tf.estimator.TrainSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          TRANSFORMED_DATA_DIR+'/train*.tfrecords',\n",
    "          batch_size=params.batch_size,\n",
    "          num_epochs=None,\n",
    "          shuffle=True\n",
    "      ),\n",
    "      max_steps=params.max_steps\n",
    "  )\n",
    "\n",
    "  eval_spec = tf.estimator.EvalSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          TRANSFORMED_DATA_DIR+'/eval*.tfrecords',\n",
    "          batch_size=params.batch_size,     \n",
    "      ),\n",
    "      start_delay_secs=0,\n",
    "      throttle_secs=0,\n",
    "      steps=None\n",
    "  )\n",
    "  \n",
    "  time_start = datetime.utcnow() \n",
    "  print(\"Experiment started at {}\".format(time_start.strftime(\"%H:%M:%S\")))\n",
    "  print(\".......................................\")\n",
    "  \n",
    "  tf.estimator.train_and_evaluate(\n",
    "    estimator=estimator,\n",
    "    train_spec=train_spec, \n",
    "    eval_spec=eval_spec)\n",
    "\n",
    "  time_end = datetime.utcnow() \n",
    "  print(\".......................................\")\n",
    "  print(\"Experiment finished at {}\".format(time_end.strftime(\"%H:%M:%S\")))\n",
    "  print(\"\")\n",
    "  \n",
    "  time_elapsed = time_end - time_start\n",
    "  print(\"Experiment elapsed time: {} seconds\".format(time_elapsed.total_seconds()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q1VBm6Bo4ntU"
   },
   "source": [
    "### 3.5 Run experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mHEsbh94n0V"
   },
   "outputs": [],
   "source": [
    "MODELS_LOCATION = 'gs://{}/flights/tfx/models/'.format(BUCKET)\n",
    "MODEL_NAME = 'dnn_classifier'\n",
    "model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)\n",
    "os.environ['MODEL_DIR'] = model_dir\n",
    "\n",
    "params = tf.contrib.training.HParams()\n",
    "params.hidden_units = [128, 64]\n",
    "params.dropout = 0.15\n",
    "params.batch_size =  128\n",
    "params.max_steps = 1000\n",
    "\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=19831006,\n",
    "    save_checkpoints_steps=200, \n",
    "    keep_checkpoint_max=3, \n",
    "    model_dir=model_dir,\n",
    "    log_step_count_steps=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 4950
    },
    "colab_type": "code",
    "id": "9yMxrzCE5tWw",
    "outputId": "25c1dead-69c2-4b14-e987-bf21b056559b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f02a4dffe80>, '_train_distribute': None, '_service': None, '_model_dir': 'gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier', '_num_ps_replicas': 0, '_log_step_count_steps': 10, '_eval_distribute': None, '_experimental_distribute': None, '_tf_random_seed': 19831006, '_master': '', '_protocol': None, '_device_fn': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_evaluation_master': '', '_num_worker_replicas': 1, '_global_id_in_cluster': 0, '_keep_checkpoint_max': 3, '_task_id': 0, '_is_chief': True, '_save_checkpoints_secs': None, '_save_summary_steps': 100, '_task_type': 'worker', '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_steps': 200}\n",
      "Removing previous artifacts...\n",
      "Experiment started at 05:31:25\n",
      ".......................................\n",
      "INFO:tensorflow:Not using Distribute Coordinator.\n",
      "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
      "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 200 or save_checkpoints_secs None.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2997: IdentityCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed after 2018-11-30.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:2703: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/feature_column/feature_column_v2.py:3828: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "INFO:tensorflow:loss = 79.157, step = 1\n",
      "INFO:tensorflow:global_step/sec: 6.69998\n",
      "INFO:tensorflow:loss = 29.836735, step = 11 (1.494 sec)\n",
      "INFO:tensorflow:global_step/sec: 160.434\n",
      "INFO:tensorflow:loss = 21.127508, step = 21 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 173.538\n",
      "INFO:tensorflow:loss = 17.358824, step = 31 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 160.065\n",
      "INFO:tensorflow:loss = 19.919691, step = 41 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 146.784\n",
      "INFO:tensorflow:loss = 28.702147, step = 51 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 165.454\n",
      "INFO:tensorflow:loss = 14.67631, step = 61 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 152.321\n",
      "INFO:tensorflow:loss = 10.69718, step = 71 (0.065 sec)\n",
      "INFO:tensorflow:global_step/sec: 156.283\n",
      "INFO:tensorflow:loss = 16.97509, step = 81 (0.064 sec)\n",
      "INFO:tensorflow:global_step/sec: 145.301\n",
      "INFO:tensorflow:loss = 14.532763, step = 91 (0.069 sec)\n",
      "INFO:tensorflow:global_step/sec: 146.69\n",
      "INFO:tensorflow:loss = 11.47735, step = 101 (0.072 sec)\n",
      "INFO:tensorflow:global_step/sec: 137.372\n",
      "INFO:tensorflow:loss = 19.939114, step = 111 (0.069 sec)\n",
      "INFO:tensorflow:global_step/sec: 149.58\n",
      "INFO:tensorflow:loss = 22.059254, step = 121 (0.067 sec)\n",
      "INFO:tensorflow:global_step/sec: 157.17\n",
      "INFO:tensorflow:loss = 9.515352, step = 131 (0.064 sec)\n",
      "INFO:tensorflow:global_step/sec: 149.376\n",
      "INFO:tensorflow:loss = 16.780128, step = 141 (0.067 sec)\n",
      "INFO:tensorflow:global_step/sec: 143.486\n",
      "INFO:tensorflow:loss = 10.5635395, step = 151 (0.070 sec)\n",
      "INFO:tensorflow:global_step/sec: 144.251\n",
      "INFO:tensorflow:loss = 6.992614, step = 161 (0.069 sec)\n",
      "INFO:tensorflow:global_step/sec: 147.179\n",
      "INFO:tensorflow:loss = 18.78405, step = 171 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 169.845\n",
      "INFO:tensorflow:loss = 8.210512, step = 181 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 174.995\n",
      "INFO:tensorflow:loss = 12.399039, step = 191 (0.058 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 200 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/metrics_impl.py:2002: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-04-02T05:31:52Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-200\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-04-02-05:33:09\n",
      "INFO:tensorflow:Saving dict for global step 200: accuracy = 0.94350755, accuracy_baseline = 0.8239612, auc = 0.9637387, auc_precision_recall = 0.99081326, average_loss = 0.16040708, global_step = 200, label/mean = 0.8239612, loss = 20.531908, precision = 0.95420724, prediction/mean = 0.83251333, recall = 0.97839135\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 200: gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-200\n",
      "INFO:tensorflow:global_step/sec: 0.111176\n",
      "INFO:tensorflow:loss = 3.565744, step = 201 (89.948 sec)\n",
      "INFO:tensorflow:global_step/sec: 13.7215\n",
      "INFO:tensorflow:loss = 7.675998, step = 211 (0.728 sec)\n",
      "INFO:tensorflow:global_step/sec: 185.757\n",
      "INFO:tensorflow:loss = 13.367186, step = 221 (0.054 sec)\n",
      "INFO:tensorflow:global_step/sec: 196.769\n",
      "INFO:tensorflow:loss = 10.163921, step = 231 (0.051 sec)\n",
      "INFO:tensorflow:global_step/sec: 191.509\n",
      "INFO:tensorflow:loss = 18.282005, step = 241 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 188.323\n",
      "INFO:tensorflow:loss = 14.06428, step = 251 (0.053 sec)\n",
      "INFO:tensorflow:global_step/sec: 190.116\n",
      "INFO:tensorflow:loss = 12.753855, step = 261 (0.055 sec)\n",
      "INFO:tensorflow:global_step/sec: 183.472\n",
      "INFO:tensorflow:loss = 18.577574, step = 271 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 186.79\n",
      "INFO:tensorflow:loss = 15.49773, step = 281 (0.054 sec)\n",
      "INFO:tensorflow:global_step/sec: 192.717\n",
      "INFO:tensorflow:loss = 16.298386, step = 291 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 185.719\n",
      "INFO:tensorflow:loss = 28.229029, step = 301 (0.055 sec)\n",
      "INFO:tensorflow:global_step/sec: 193.931\n",
      "INFO:tensorflow:loss = 18.915554, step = 311 (0.051 sec)\n",
      "INFO:tensorflow:global_step/sec: 154.29\n",
      "INFO:tensorflow:loss = 18.243713, step = 321 (0.065 sec)\n",
      "INFO:tensorflow:global_step/sec: 192.603\n",
      "INFO:tensorflow:loss = 20.666748, step = 331 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 170.962\n",
      "INFO:tensorflow:loss = 14.940548, step = 341 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 193.234\n",
      "INFO:tensorflow:loss = 21.147757, step = 351 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 191.118\n",
      "INFO:tensorflow:loss = 13.885234, step = 361 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 205.755\n",
      "INFO:tensorflow:loss = 20.689209, step = 371 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 209.884\n",
      "INFO:tensorflow:loss = 14.330908, step = 381 (0.047 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.433\n",
      "INFO:tensorflow:loss = 28.152597, step = 391 (0.048 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 400 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-04-02T05:33:23Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-400\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-04-02-05:34:39\n",
      "INFO:tensorflow:Saving dict for global step 400: accuracy = 0.9446056, accuracy_baseline = 0.8239612, auc = 0.96578443, auc_precision_recall = 0.99105406, average_loss = 0.15215103, global_step = 400, label/mean = 0.8239612, loss = 19.475143, precision = 0.9566209, prediction/mean = 0.82234436, recall = 0.97707736\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 400: gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-400\n",
      "INFO:tensorflow:global_step/sec: 0.116538\n",
      "INFO:tensorflow:loss = 11.816775, step = 401 (86.315 sec)\n",
      "INFO:tensorflow:global_step/sec: 7.82218\n",
      "INFO:tensorflow:loss = 24.750416, step = 411 (0.772 sec)\n",
      "INFO:tensorflow:global_step/sec: 219.924\n",
      "INFO:tensorflow:loss = 32.50969, step = 421 (0.045 sec)\n",
      "INFO:tensorflow:global_step/sec: 209.571\n",
      "INFO:tensorflow:loss = 27.117569, step = 431 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.758\n",
      "INFO:tensorflow:loss = 20.522116, step = 441 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 197.676\n",
      "INFO:tensorflow:loss = 35.40588, step = 451 (0.050 sec)\n",
      "INFO:tensorflow:global_step/sec: 211.918\n",
      "INFO:tensorflow:loss = 28.383389, step = 461 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 202.109\n",
      "INFO:tensorflow:loss = 33.852127, step = 471 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.661\n",
      "INFO:tensorflow:loss = 18.847904, step = 481 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 210.331\n",
      "INFO:tensorflow:loss = 25.394344, step = 491 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 213.257\n",
      "INFO:tensorflow:loss = 27.13015, step = 501 (0.047 sec)\n",
      "INFO:tensorflow:global_step/sec: 208.449\n",
      "INFO:tensorflow:loss = 24.80013, step = 511 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 201.458\n",
      "INFO:tensorflow:loss = 24.043844, step = 521 (0.050 sec)\n",
      "INFO:tensorflow:global_step/sec: 211.706\n",
      "INFO:tensorflow:loss = 26.673195, step = 531 (0.047 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.135\n",
      "INFO:tensorflow:loss = 36.733288, step = 541 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 167.306\n",
      "INFO:tensorflow:loss = 35.161667, step = 551 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 196.046\n",
      "INFO:tensorflow:loss = 26.572227, step = 561 (0.051 sec)\n",
      "INFO:tensorflow:global_step/sec: 193.875\n",
      "INFO:tensorflow:loss = 17.721455, step = 571 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 211.99\n",
      "INFO:tensorflow:loss = 32.65547, step = 581 (0.047 sec)\n",
      "INFO:tensorflow:global_step/sec: 211.503\n",
      "INFO:tensorflow:loss = 30.736437, step = 591 (0.047 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 600 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to delete files with this prefix.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-04-02T05:34:52Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-600\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-04-02-05:36:07\n",
      "INFO:tensorflow:Saving dict for global step 600: accuracy = 0.9430892, accuracy_baseline = 0.8239612, auc = 0.96750224, auc_precision_recall = 0.9915764, average_loss = 0.15369783, global_step = 600, label/mean = 0.8239612, loss = 19.673134, precision = 0.9614269, prediction/mean = 0.8089146, recall = 0.9698408\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 600: gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-600\n",
      "INFO:tensorflow:global_step/sec: 0.115921\n",
      "INFO:tensorflow:loss = 24.700073, step = 601 (86.267 sec)\n",
      "INFO:tensorflow:global_step/sec: 14.8469\n",
      "INFO:tensorflow:loss = 38.352432, step = 611 (0.673 sec)\n",
      "INFO:tensorflow:global_step/sec: 195.652\n",
      "INFO:tensorflow:loss = 15.268581, step = 621 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 193.589\n",
      "INFO:tensorflow:loss = 21.598988, step = 631 (0.051 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.908\n",
      "INFO:tensorflow:loss = 19.122883, step = 641 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 200.025\n",
      "INFO:tensorflow:loss = 26.627052, step = 651 (0.050 sec)\n",
      "INFO:tensorflow:global_step/sec: 189.858\n",
      "INFO:tensorflow:loss = 29.231117, step = 661 (0.053 sec)\n",
      "INFO:tensorflow:global_step/sec: 200.257\n",
      "INFO:tensorflow:loss = 22.25672, step = 671 (0.050 sec)\n",
      "INFO:tensorflow:global_step/sec: 185.285\n",
      "INFO:tensorflow:loss = 34.08339, step = 681 (0.053 sec)\n",
      "INFO:tensorflow:global_step/sec: 202.804\n",
      "INFO:tensorflow:loss = 24.448158, step = 691 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 215.39\n",
      "INFO:tensorflow:loss = 28.652527, step = 701 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.725\n",
      "INFO:tensorflow:loss = 29.50911, step = 711 (0.047 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.479\n",
      "INFO:tensorflow:loss = 34.088882, step = 721 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 200.785\n",
      "INFO:tensorflow:loss = 36.38434, step = 731 (0.050 sec)\n",
      "INFO:tensorflow:global_step/sec: 209.834\n",
      "INFO:tensorflow:loss = 27.82328, step = 741 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 203.5\n",
      "INFO:tensorflow:loss = 19.732765, step = 751 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 185.026\n",
      "INFO:tensorflow:loss = 26.701561, step = 761 (0.054 sec)\n",
      "INFO:tensorflow:global_step/sec: 204.663\n",
      "INFO:tensorflow:loss = 18.516378, step = 771 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 174.645\n",
      "INFO:tensorflow:loss = 30.306314, step = 781 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 192.88\n",
      "INFO:tensorflow:loss = 22.32777, step = 791 (0.052 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 800 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-04-02T05:36:20Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-800\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-04-02-05:37:38\n",
      "INFO:tensorflow:Saving dict for global step 800: accuracy = 0.9457019, accuracy_baseline = 0.8239612, auc = 0.9676392, auc_precision_recall = 0.9916347, average_loss = 0.14911827, global_step = 800, label/mean = 0.8239612, loss = 19.086954, precision = 0.95805603, prediction/mean = 0.82110894, recall = 0.97686875\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 800: gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-800\n",
      "INFO:tensorflow:global_step/sec: 0.111582\n",
      "INFO:tensorflow:loss = 22.332035, step = 801 (90.082 sec)\n",
      "INFO:tensorflow:global_step/sec: 19.3722\n",
      "INFO:tensorflow:loss = 24.977993, step = 811 (0.055 sec)\n",
      "INFO:tensorflow:global_step/sec: 13.8203\n",
      "INFO:tensorflow:loss = 28.424976, step = 821 (0.724 sec)\n",
      "INFO:tensorflow:global_step/sec: 171.704\n",
      "INFO:tensorflow:loss = 24.512535, step = 831 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 201.697\n",
      "INFO:tensorflow:loss = 28.081326, step = 841 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 206.1\n",
      "INFO:tensorflow:loss = 28.34459, step = 851 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 200.446\n",
      "INFO:tensorflow:loss = 24.169115, step = 861 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 180.535\n",
      "INFO:tensorflow:loss = 28.302261, step = 871 (0.053 sec)\n",
      "INFO:tensorflow:global_step/sec: 202.829\n",
      "INFO:tensorflow:loss = 33.0009, step = 881 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 203.827\n",
      "INFO:tensorflow:loss = 31.68254, step = 891 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.96\n",
      "INFO:tensorflow:loss = 30.916084, step = 901 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 205.396\n",
      "INFO:tensorflow:loss = 22.248018, step = 911 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 204.031\n",
      "INFO:tensorflow:loss = 21.602924, step = 921 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 207.994\n",
      "INFO:tensorflow:loss = 19.120996, step = 931 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 201.715\n",
      "INFO:tensorflow:loss = 31.52656, step = 941 (0.049 sec)\n",
      "INFO:tensorflow:global_step/sec: 202.036\n",
      "INFO:tensorflow:loss = 26.380928, step = 951 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 199.814\n",
      "INFO:tensorflow:loss = 19.218407, step = 961 (0.048 sec)\n",
      "INFO:tensorflow:global_step/sec: 190.81\n",
      "INFO:tensorflow:loss = 30.414865, step = 971 (0.052 sec)\n",
      "INFO:tensorflow:global_step/sec: 189.418\n",
      "INFO:tensorflow:loss = 30.230112, step = 981 (0.053 sec)\n",
      "INFO:tensorflow:global_step/sec: 194.673\n",
      "INFO:tensorflow:loss = 30.989592, step = 991 (0.051 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-04-02T05:37:51Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-04-02-05:39:06\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.94401014, accuracy_baseline = 0.8239612, auc = 0.96807283, auc_precision_recall = 0.9917144, average_loss = 0.15245882, global_step = 1000, label/mean = 0.8239612, loss = 19.51454, precision = 0.96206194, prediction/mean = 0.80586094, recall = 0.9703113\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/model.ckpt-1000\n",
      "INFO:tensorflow:Loss for final step: 25.53121.\n",
      ".......................................\n",
      "Experiment finished at 05:39:08\n",
      "\n",
      "Experiment elapsed time: 463.192468 seconds\n"
     ]
    }
   ],
   "source": [
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pr-L_KM8LwmK"
   },
   "source": [
    "### 3.6 Export the model for serving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oBdB7mkvL43C"
   },
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "RAW_SCHEMA_LOCATION = 'raw_schema.pbtxt'\n",
    "def make_serving_input_receiver_fn():\n",
    "  from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "  source_raw_schema = tfdv.load_schema_text(RAW_SCHEMA_LOCATION)\n",
    "  raw_feature_spec = schema_utils.schema_as_feature_spec(source_raw_schema).feature_spec\n",
    "  raw_feature_spec.pop(TARGET_FEATURE_NAME)\n",
    "\n",
    "  # Create the interface for the serving function with the raw features\n",
    "  raw_features = tf.estimator.export.build_parsing_serving_input_receiver_fn(raw_feature_spec)().features\n",
    "\n",
    "  receiver_tensors = {feature: tf.placeholder(shape=[None], dtype=raw_features[feature].dtype) \n",
    "    for feature in raw_features\n",
    "  }\n",
    "\n",
    "  receiver_tensors_expanded = {tensor: tf.reshape(receiver_tensors[tensor], (-1, 1)) \n",
    "    for tensor in receiver_tensors\n",
    "  }\n",
    "\n",
    "  # Apply the transform function \n",
    "  transformed_features = transform_output.transform_raw_features(receiver_tensors_expanded)\n",
    "\n",
    "  return tf.estimator.export.ServingInputReceiver(\n",
    "    transformed_features, receiver_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 751
    },
    "colab_type": "code",
    "id": "_KcSJ6kEMG57",
    "outputId": "d0bcdb14-82ef-4b22-9484-90170f1b7984"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/export/1554183658'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "\n",
    "if tf.gfile.Exists(export_dir):\n",
    "    tf.gfile.DeleteRecursively(export_dir)\n",
    "        \n",
    "estimator.export_savedmodel(\n",
    "    export_dir_base=export_dir,\n",
    "    serving_input_receiver_fn=make_serving_input_receiver_fn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1377
    },
    "colab_type": "code",
    "id": "XPFCTtSdQMd5",
    "outputId": "2ec32f67-47f5-4ef0-8a86-fe1797befdef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/export/1554183658/\n",
      "\n",
      "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
      "\n",
      "signature_def['predict']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['arr_lat'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_2:0\n",
      "    inputs['arr_lon'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_4:0\n",
      "    inputs['avg_arr_delay'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_10:0\n",
      "    inputs['avg_dep_delay'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_9:0\n",
      "    inputs['carrier'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1)\n",
      "        name: Placeholder_6:0\n",
      "    inputs['dep_delay'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder:0\n",
      "    inputs['dep_lat'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_1:0\n",
      "    inputs['dep_lon'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_11:0\n",
      "    inputs['dest'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1)\n",
      "        name: Placeholder_5:0\n",
      "    inputs['distance'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_3:0\n",
      "    inputs['origin'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1)\n",
      "        name: Placeholder_7:0\n",
      "    inputs['taxiout'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1)\n",
      "        name: Placeholder_8:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['class_ids'] tensor_info:\n",
      "        dtype: DT_INT64\n",
      "        shape: (-1, 1)\n",
      "        name: dnn/head/predictions/ExpandDims:0\n",
      "    outputs['classes'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1, 1)\n",
      "        name: dnn/head/predictions/str_classes:0\n",
      "    outputs['logistic'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 1)\n",
      "        name: dnn/head/predictions/logistic:0\n",
      "    outputs['logits'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 1)\n",
      "        name: dnn/logits/BiasAdd:0\n",
      "    outputs['probabilities'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 2)\n",
      "        name: dnn/head/predictions/probabilities:0\n",
      "  Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "saved_models_base=${MODEL_DIR}/export/\n",
    "saved_model_dir=$(gcloud storage ls ${saved_models_base} | tail -n 1)\n",    "echo ${saved_model_dir}\n",
    "saved_model_cli show --dir=${saved_model_dir} --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFQcy8QDQ6Ad"
   },
   "source": [
    "### 3.7 Try out saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "0tacIgD5Q6HB",
    "outputId": "133588d7-1d0b-4a9f-fd88-b79d096f55b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/export/1554183658/\n",
      "\n",
      "{'dep_delay': [14.0], 'dep_lat': [32.85], 'arr_lat': [31.94], 'dest': ['MAF'], 'arr_lon': [-102.2], 'distance': [319.0], 'carrier': ['WN'], 'origin': ['DAL'], 'taxiout': [13.0], 'avg_dep_delay': [25.86], 'avg_arr_delay': [27.0], 'dep_lon': [-96.85]}\n",
      "\n",
      "{'logits': array([[-1.2914119]], dtype=float32), 'probabilities': array([[0.78438604, 0.21561392]], dtype=float32), 'class_ids': array([[0]]), 'logistic': array([[0.21561392]], dtype=float32), 'classes': array([[b'0']], dtype=object)}\n"
     ]
    }
   ],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "tf.gfile.ListDirectory(export_dir)[-1]\n",
    "saved_model_dir = os.path.join(export_dir, tf.gfile.ListDirectory(export_dir)[-1])\n",
    "print(saved_model_dir)\n",
    "print()\n",
    "\n",
    "predictor_fn = tf.contrib.predictor.from_saved_model(\n",
    "    export_dir = saved_model_dir,\n",
    "    signature_def_key=\"predict\"\n",
    ")\n",
    "\n",
    "input = {\n",
    "        'dep_delay': [14.0],\n",
    "        'taxiout': [13.0],\n",
    "        'distance': [319.0],\n",
    "        'avg_dep_delay': [25.86],\n",
    "        'avg_arr_delay': [27.0],\n",
    "        'carrier': ['WN'],\n",
    "        'dep_lat': [32.85],\n",
    "        'dep_lon': [-96.85],\n",
    "        'arr_lat': [31.94],\n",
    "        'arr_lon': [-102.2], \n",
    "        'origin': ['DAL'], \n",
    "        'dest': ['MAF']\n",
    "}\n",
    "\n",
    "print(input)\n",
    "print()\n",
    "output = predictor_fn(input)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.8 Deploy model to Cloud ML Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "MODEL_NAME=\"flights\"\n",
    "MODEL_VERSION=\"tfx\"\n",
    "MODEL_LOCATION=$(gcloud storage ls gs://${BUCKET}/flights/tfx/models/dnn_classifier/export/ | tail -1)\n",    "#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION\n",
    "#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version 1.13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"dep_delay\": 14.0,\"taxiout\": 13.0,\"distance\": 319.0,\"avg_dep_delay\": 25.86,\"avg_arr_delay\": 27.0, \"carrier\": \"WN\",\"dep_lat\": 32.85,\"dep_lon\": -96.85,\"arr_lat\": 31.94,\"arr_lon\": -102.2, \"origin\": \"DAL\", \"dest\": \"MAF\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud ml-engine predict --model=flights --version=tfx --json-instances input.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.9 Export evaluation saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_eval_input_receiver_fn():\n",
    "  receiver_tensors = {'examples': tf.placeholder(dtype=tf.string, shape=[None])}\n",
    "  columns = tf.decode_csv(receiver_tensors['examples'], record_defaults=DEFAULTS)\n",
    "  features = dict(zip(CSV_COLUMNS, columns))\n",
    "  print(features)\n",
    "\n",
    "  for feature_name in features:\n",
    "    if features[feature_name].dtype == tf.int32:\n",
    "      features[feature_name] = tf.cast(features[feature_name], tf.int64)\n",
    "    features[feature_name] = tf.reshape(features[feature_name], (-1, 1))\n",
    "\n",
    "  transformed_features = transform_output.transform_raw_features(features)\n",
    "  features.update(transformed_features)\n",
    "\n",
    "  return tfma.export.EvalInputReceiver(\n",
    "    features=features,\n",
    "    receiver_tensors=receiver_tensors,\n",
    "    labels=features[TARGET_FEATURE_NAME]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'dep_delay': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=float32>, 'dep_lat': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=float32>, 'arr_lat': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=float32>, 'dest': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=string>, 'arr_lon': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=float32>, 'distance': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=float32>, 'avg_arr_delay': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=float32>, 'carrier': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=string>, 'origin': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=string>, 'taxiout': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=float32>, 'avg_dep_delay': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=float32>, 'ontime': <tf.Tensor 'DecodeCSV:0' shape=(?,) dtype=float32>, 'dep_lon': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=float32>}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "b'gs://cloud-training-demos-ml/flights/tfx/models/dnn_classifier/export/evaluate/1554184743'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow_model_analysis as tfma\n",
    "eval_model_dir = os.path.join(model_dir, \"export/evaluate\")\n",
    "if tf.gfile.Exists(eval_model_dir):\n",
    "    tf.gfile.DeleteRecursively(eval_model_dir)\n",
    "\n",
    "tfma.export.export_eval_savedmodel(\n",
    "        estimator=estimator,\n",
    "        export_dir_base=eval_model_dir,\n",
    "        eval_input_receiver_fn=make_eval_input_receiver_fn\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gsi_Hsh89Cl7"
   },
   "source": [
    "## License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0fOWx1yI9Dyn"
   },
   "source": [
    "Copyright 2019 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "\n",
    "---\n",
    "This is not an official Google product. The sample code provided for educational purposes only.\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "02-tfx_end_to_end",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
